Super-resolution in processing images such as from multi-layer sensors

ABSTRACT

Pixels of an array capture values for an input image. The captured values correspond to pixel center points of the pixels that captured the values. Additional values about the input image may be further computed from the captured values. These additional values may correspond to additional center points, which can be different from any of the pixel center points. An output image may be constructed from the captured values plus the additional values. These values may be stored together and/or displayed together as the output image. Embodiments are applicable to images such as from multi-layer sensors. Since the output image can be created from a higher total number of values than those captured by the available number of pixels, such image processing can be called super-resolution, and can be applied to image processing within imaging devices, for still pictures, video and motion pictures, and so on.

CROSS REFERENCE TO RELATED PATENT APPLICATIONS

This patent application claims priority from U.S. Provisional PatentApplication Ser. No. 61/897,773, filed on Oct. 30, 2013, the disclosureof which is hereby incorporated by reference for all purposes.

BACKGROUND

Modern imaging devices use pixels to capture images. The pixels dividean input image in elements, and capture values for the elements of theimage. These values for the image are captured by various techniques,such as numbers of electrons per pixel after a brief exposure time. Theoutput image is typically constructed from the captured values, whetherin color or in black and white.

BRIEF SUMMARY

The present description gives instances of devices, computers andmethods, the use of which may help overcome problems and limitations ofthe prior art.

In some embodiments, pixels of an array may capture values for an inputimage. The captured values may correspond to pixel center points of therespective pixels that captured the values. One or more additionalvalues about the input image may be further computed from the capturedvalues. At least some of these additional values correspond toadditional center points, which can be different from any of the pixelcenter points. An output image may be constructed from the capturedvalues, together with the additional values. These values may be storedtogether and/or displayed together as the output image.

An advantage of embodiments arises from the fact that the output imagecan be created from a higher total number of values than those capturedby the available number of pixels. Such image processing can be calledsuper-resolution, and can be applied to image processing within imagingdevices, for still pictures, video and motion pictures, and so on.

These and other features and advantages of this description will becomemore readily apparent from the following Detailed Description, whichproceeds with reference to the drawings, in which:

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an imaging device made according toembodiments.

FIG. 2 is a diagram of a sample pixel array with additional samplechosen locations for computation of additional values, according toembodiments.

FIG. 3 is a diagram of a computer according to embodiments.

FIG. 4 is a flowchart illustrating methods according to embodiments.

FIG. 5 is a diagram of a tile of a possible single-layer sensorembodiment of the pixel array of FIG. 2.

FIG. 6 is a diagram of a tile pair of a possible two-layer stackedsensor array made according to embodiments.

FIG. 7 is a diagram of four tiles of a possible three-layer stackedsensor array made according to embodiments.

FIG. 8 is a diagram of a two-layer offset stacked sensor array portionmade according to embodiments.

FIG. 9 is a diagram of a two-layer offset stacked sensor array portionmade according to a rotated embodiment.

FIG. 10 is a diagram of a three-layer offset stacked sensor arrayportion made according to embodiments.

FIG. 11 is a diagram of a three-layer offset stacked sensor arrayportion made according to a rotated embodiment.

FIG. 12 is a diagram of a three-layer offset stacked sensor arrayportion made according to embodiments.

FIG. 13 depicts a controller-based system for an imaging device, whichuses an imaging array made according to embodiments.

FIG. 14 is a diagram of a physical two-layer offset stacked sensor arrayportion, according to embodiments.

FIG. 15 is a diagram of a portion of a physical sample imageconstruction grid according to embodiments.

FIG. 16 is a diagram of a portion of a sample image construction gridaccording to embodiments.

FIG. 17 is a diagram showing a single-layer sensor array portion, andfurther identifying a sample tile for locating additional center pointsfor additional values according to embodiments.

FIG. 18 is a diagram showing a three-layer stacked sensor array portion,and further identifying a sample tile for locating additional centerpoints for additional values according to embodiments.

FIG. 19 is a diagram showing a two-layer stacked sensor array portion,and further identifying sample tiles for locating additional centerpoints for additional values according to embodiments.

FIG. 20 is a diagram showing a two-layer stacked offset sensor arrayportion, and further identifying a sample tile for locating additionalcenter points for additional values according to embodiments.

FIG. 21 shows sequential diagrams depicting another journey from thearray portion of FIG. 20 to an image construction grid according toembodiments.

FIG. 22 shows sequential diagrams depicting gathering captured valuesfor computing an additional value for an additional center point in thearray portion of FIG. 14, according to embodiments.

FIG. 23 shows sequential diagrams depicting gathering captured valuesfor computing an additional value for a different additional centerpoint in the array portion of FIG. 14, according to embodiments.

FIG. 24 shows equations that can be used for computing R, G, B imagevalues according to embodiments.

FIGS. 25A, 25B, 25C and 25D show equations that can be used forcomputing luminance and chrominance image values according toembodiments.

FIG. 26 is a diagram for describing filter training operations accordingto embodiments.

FIG. 27 is a flowchart for illustrating filter training methodsaccording to embodiments.

FIG. 28 shows equations that can be used for training filters directlywith RGB values according to embodiments.

FIGS. 29A, 29B, 29C, 29D show equations that can be used for trainingfilters first by in luminance and chrominance values according toembodiments.

FIG. 30 is a diagram illustrating image operations according toembodiments.

FIG. 31 is a diagram illustrating resizing of an output image from theinput image according to embodiments.

FIG. 32 is a diagram illustrating additional composite image operationsaccording to embodiments.

FIG. 33 is a diagram illustrating a specific composite image operationaccording to embodiments.

DETAILED DESCRIPTION

As has been mentioned, the present description is about devices,computers and methods that can perform certain types of imageprocessing. Embodiments are now described in more detail.

FIG. 1 is a block diagram of an imaging device 100 made according toembodiments. Imaging device 100 has a casing 102. A light source 105,such as an LED, may be provided on casing 102. An opening OP is providedin casing 102. A lens LN may be provided optionally at opening OP,although that is not necessary.

Imaging device 100 also has a pixel array 110 made according toembodiments. Pixel array 110 is configured to receive light throughopening OP, so imaging device 100 can capture an image of an object OBJ,person, or scene. Sometimes, the image capture is assisted by lightsource 105. As can be seen, pixel array 110 and opening OP define anominal Field of View FOV-N. Of course, Field of View FOV-N and objectOBJ are in three dimensions, while FIG. 1 shows them in two dimensions.Further, if lens LN is indeed provided, the resulting actual field ofview may be different than nominal Field of View FOV-N. Imaging device100 is aligned so that object OBJ, person, or scene that is to be imagedis brought within the actual field of view.

The pixels of pixel array 110 can capture elements of the image. In manyembodiments, pixel array 110 has a two-dimensional array of pixels. Thearray can be organized in rows and columns.

Device 100 can render the image from the elements captured by thepixels. Optionally, device 100 also includes a display 180, which caninclude a screen or a touchscreen that can display the rendered image,or a version of it.

Device 100 additionally includes a controller 120, for controlling theoperation of pixel array 110 and other components of imaging device 100.Controller 120 may optionally be formed integrally with pixel array 110,and possibly also with other components of imaging device 100.

Device 100 may additionally include an image processor 125, forprocessing the input image according to embodiments. More particularly,image processor 125 may process captured values by the pixels of array110 of the input image, to generate the values that constitute theoutput image. Image processor 125 may optionally be formed integrallywith controller 120, pixel array 110, and/or other components of imagingdevice 100.

Device 100 may optionally include a memory 140. Memory 140 may store theinput image, i.e. the values captured by the pixels of the array, forthe respective elements of the input image. Memory 140 may further storeaspects of the output image, e.g. values computed for the output imageaccording to embodiments.

FIG. 2 is a diagram of a sample pixel array 210. Pixel array 210 haspixels 232, of which only a few are shown, and even fewer are indicated,for clarity. Pixels 232 can be made from inorganic material and/ororganic material.

Pixels 232 are shown as squares, but that is only for purposes ofdepiction. In reality, pixels 232 can have different shapes, and theirphotosensitive area can be less than the area of the entire pixel.Accordingly, the photosensitive areas of pixels 232, taken together, donot necessarily cover the entire area of array 210. In fact, in someembodiments, elements of an input image that are imaged between thephotosensitive areas of pixels 232 might not be imaged, which means thatvalues for such elements might not be captured.

Pixels 232 are arranged in a rectangular pattern along rows 222 andcolumns 224, but that is only for purposes of example and not oflimitation. As will be seen in other examples later in this document,there can be multiple patterns of arranging the pixels.

In FIG. 2, only one layer of pixels 232 is shown, but that is notnecessarily the case. As will be seen in other examples later in thisdocument, there can be multiple layers, such as two or more layers ofpixels, which could be stacked. A pair of such layers can typically becharacterized as a lower layer and an upper layer that is verticallystacked over the lower layer, whether there is an intervening layer ornot.

Also, pixel center points 236 can be defined for pixels 232. Moreparticularly, pixel center points 236 can be defined in a suitable,preferably consistent way, such as being at the center of thephotosensitive areas of pixels 232. If these photosensitive areas, alsocalled photo-sites, are rectangular, for example, the center can befound at the intersection of two diagonals of the photosensitive area.In this example, rows 222 and columns 224 were advantageously chosensuch that pixel center points 236 are located at their intersections.For the mathematical purposes of constructing the output image, thevalues captured by pixels 232 may correspond to pixel center points 236of the respective pixels 232 that captured them.

Moreover, in FIG. 2, additional center points 239 can be further chosen.In the example of FIG. 2, none of the shown additional center points 239is at the intersections of rows 222 and columns 224, but that is notnecessary.

As mentioned previously, in embodiments, additional values are computedfor the output image, above and beyond the captured values. Suchadditional values may correspond to additional center points 239. Sinceadditional center points 239 are not at the intersections of rows 222and columns 224, these additional values may thus be for additionalelements of the input image that may not have been captured by one ofpixels 232. Therefore, when one of the additional values is computed forthe input image according to embodiments, that additional value can beused to help construct a final output image above and beyond the valuescaptured for the input image by physical pixels 232.

In FIG. 2 additional center points 236 are drawn as full dots, whileadditional center points 239 are drawn as circles, differently. Thedifference is intended to help keep in mind that pixel center points 236are defined for values that are captured by pixels 232, while additionalcenter points 239 are defined for additional values that are computed.The output image may then have values for locations with the dots, andalso for locations with the circles.

Beyond the example of FIG. 2, one or more of the chosen additionalcenter points 239 could coincide with one of pixel center points 236.Such would be useful, for example, for correcting for a bad pixel.

As mentioned above, according to embodiments, additional values arecomputed that may correspond to the additional center points. Moredetailed examples of such computations are shown later in this document.Computation may take place by an image processor, such as imageprocessor 125. Alternately the computation may be performed by acomputer. An example is now described.

FIG. 3 is a diagram of a computer 310 according to embodiments. Computer310 may be an embodiment that receives one or more input images, andgenerates output images. The input images may be unrelated, or related,such as those of a video or a moving picture.

Computer 310 may be implemented by one or more devices that includelogic circuitry. Computer 310 may be a standalone device or computer,such as a general purpose computer, or part of a device that has one ormore additional functions. The logic circuitry may include a processor320 that may be programmable for a general purpose, or dedicated, suchas a microcontroller, a microprocessor, a Digital Signal Processor(DSP), etc. The logic circuitry may also include storage media, such asa memory 330. Such media include but are not limited to volatile memory,non-volatile memory (NVM), read only memory (ROM); random access memory(RAM); magnetic disk storage media; optical storage media; smart cards,flash memory devices, etc. Any one of these storage media can be anon-transitory computer-readable medium. These storage media,individually or in combination with others, can have stored thereon data334. In addition, these storage media may store programs 332 thatprocessor 320 may be able to read, and execute. More particularly,programs 332 can include instructions in the form of code, whichprocessor 320 may be able to execute upon reading. Executing isperformed by physical manipulations of physical quantities, and mayresult in the functions, processes, actions and/or methods to beperformed, and/or processor 320 to cause other devices or components orblocks to perform such functions, processes, actions and/or methods.

Returning to FIG. 1, if memory 140 is indeed provided, it can be one ormore memories. The one or more memories can be configured to store thecaptured values and the additional values, for later export or display.Both sets of values can be used to contribute to a single output image.In embodiments, therefore, a sum of the number of the captured valuesplus the number of the additional values is larger than the number ofthe pixels in the array. That is because the number of the capturedvalues generally may equal the number of the pixels, but the additionalvalues simply provide more data.

In some embodiments, the additional values are further merged with thecaptured values when stored. For purposes of this document, this mergingmeans that the additional values have been intermingled with thecaptured values for uniformly executed export or display according totheir location for the output image; and that they are not stored inseparate memories or in separate memory sections.

Additionally, if display 180 is provided, it can be configured todisplay an output image, from the input image. The output image can beconstructed from the values captured by the pixels, and from theadditional computed values.

The devices and/or systems described in this document perform functions,processes and/or methods, as described in this document. Moreover,methods and algorithms are described below. These methods and algorithmsare not necessarily inherently associated with any particular logicdevice or other apparatus. Rather, they are advantageously implementedby programs for use by a computing machine, such as a general-purposecomputer, a special purpose computer, a microprocessor, etc.

Often, for the sake of convenience only, it is preferred to implementand describe a program as various interconnected distinct softwaremodules or features, individually and collectively also known assoftware. This is not necessary, however, and there may be cases wheremodules are equivalently aggregated into a single program, even withunclear boundaries. In some instances, software is combined withhardware, in a mix called firmware.

This detailed description includes flowcharts, display images,algorithms, and symbolic representations of program operations within atleast one computer readable medium. An economy is achieved in that asingle set of flowcharts is used to describe both programs, and alsomethods. So, while flowcharts described methods in terms of boxes, theyalso concurrently describe programs.

Methods are now described.

FIG. 4 shows a flowchart 400 for describing methods according toembodiments. The methods of flowchart 400 may also be practiced byembodiments described in this document.

According to an operation 410, captured values may be received. Thecaptured values can be values of respective elements of an input image,and may have been captured by pixels. The captured values may correspondto pixel center points of respective ones of the pixels that capturedthe values.

According to another operation 430, additional values are computed fromthe captured values. The additional values may correspond to additionalcenter points. At least some of the additional elements might bedifferent from the pixel center points, in other words, not coincidewith them.

An output image can thus be formed, from the captured values and thecomputed values. In the output image, the captured values may correspondto the pixel center points, and the additional values may correspond tothe additional center points.

According to another, optional operation 440, the captured values andthe additional values are caused to be stored. Storing can be in one ormore memories.

According to another, optional operation 450, an output image isdisplayed, or caused to be displayed. The output image may have beenconstructed from the captured values and from the additional values.

Returning to FIG. 2, pixels 232 can be black and white, or color. Ifthey are black and white, then each pixel could be similar to theothers. If pixels 232 are color, they could be, for example Red GreenBlue, or other combination of color pixels. When operating in such acombination, a pixel may be different from its neighbor. The minimumgroup of pixels that can operate in such a combination can be called atile. A pixel array can be considered to be made of substantiallysimilar, or substantially identical tiles.

Examples will now be described of pixel arrays that can capture values,and for which additional values can be computed according toembodiments. In these, examples of tiles will also be shown.

FIG. 5 is a diagram of a tile 511. Tile 511 is a tile of a possiblesingle-layer sensor embodiment of the pixel array of FIG. 2. Tile 511includes two G pixels, one B pixel and one R pixel, all arranged in aBayer pattern. The pixels have pixel center points 536, shown as dots.The pixels of tile 511 correspond to pixel center points 536, for themathematical purposes of constructing the output image.

The pixels of tile 511 capture values for the input image. Plus, inembodiments, additional values can be computed about the input image.These additional values could correspond to additional center points539, for the mathematical purposes of constructing the output image. Aswill be seen, such additional center points could be defined alsoelsewhere with respect to tile 511.

FIG. 6 is a diagram of a tile pair 611 of pixels, which are also knownas sensors. Tile pair 611 is a pair of tiles of a possible two-layerpixel embodiment. The upper layer has green pixels 645. The lower layerhas blue pixels 644 alternating with red pixels 647, thus effectivelycreating a Bayer pattern. A tile in pair 611 is made from a combinationof any two green pixels 645 that have beneath them a blue pixel 644 anda red pixel 647. So, a single tile can be horizontally the top twopixels, the two pixels on the left, and so on.

In FIG. 6, pixel center points 636 are shown. A pixel center point 636is the same for one of the green pixels 645 and the one underneath it,because in this embodiment the layers are stacked without an offset. Inother words, center points of the photoreceptive area of pixels in theupper layer are vertically aligned with center points of thephotoreceptive area of pixels in the lower layer.

The pixels of tile pair 611 capture values for the input image. Plus, inembodiments, additional values can be computed about the input image.These additional values could correspond to additional center points639, for the mathematical purposes of constructing the output image. Forany one of the shown additional center points 639, additional values canbe computed for the same location with reference to more tiles in thearray, even substantially all of the tiles.

FIG. 7 is a diagram of four tiles 711 of pixels. Tiles 711 are from apossible three-layer pixel embodiment. The upper layer has green pixels745, the middle layer has blue pixels 744, and the lower layer has redpixels 747. These sensors do not form a Bayer pattern. A tile in FIG. 7is made from any green pixel 745, and those pixels underneath it.

In FIG. 7, pixel center points 736 are shown. A pixel center point 736is the same for one of the green pixels 745 and the ones underneath it,because in this embodiment the layers are again stacked without anoffset.

The pixels of tiles 711 capture values for the input image. Plus, inembodiments, additional values can be computed about the input image.The additional values could correspond to additional center points 739,for the mathematical purposes of constructing the output image. For anyone of the shown additional center points 739, additional values can becomputed for the same location with reference to more tiles in thearray, even substantially all of the tiles.

The previous two examples used stacked layers, but the pixels had nooffset. Embodiments can also have stacked pixels whose photoreceptorareas are offset from each other. In such cases, the pixel center pointscan be defined by vertically merging the planes of the two layers. Moreanalytically, the pixel center points can be defined as those for thepixels in the upper layer, plus those for projections of the pixels inthe lower layer onto a plane of the upper layer. Of course, suchprojections are intended to be vertically in the geometric sense. Anexample is now described.

FIG. 8 is a diagram of a two-layer offset stacked sensor array portion811. Array portion 811 includes an upper layer of green pixels 845, anda lower layer of red pixels 847 alternating with blue pixels 844. Thepixels of array portion 811 have pixel center points 836, indicated asfull dots. Pixel center points 836 may correspond to locations ofcaptured values. It will be appreciated that, since the plane of thediagram collapses the plane of the lower layer and the upper layer,pixel center points 836 are in the same plane regardless of whether theywere contributed by pixels of the upper layer or of the lower layer.

The center points of pixels 845 in the upper layer are disposed alongrows 822 and columns 824. Rows 822 have a distance 862 between them,which is called the vertical pixel pitch. Columns 824 have a distance884 between them, which is called the horizontal pixel pitch. In thisembodiment, the vertical pitch equals the horizontal pitch, althoughthat is not necessary.

The center points of pixels 847 and 844 in the lower layer are disposedalong rows, such as row 852, and columns, such as column 854. In manyembodiments, the rows in the lower layer have a distance between themsimilar to the vertical pixel pitch, and the columns in the lower layerhave a distance between them similar to the horizontal pixel pitch.

In the particular example of FIG. 8 the horizontal pixel pitch and thevertical pixel pitch are the same, PTC. The dimension PTC is also calledthe pixel pitch. The lower layer is offset from the upper layer,vertically by a distance 863 equaling PTC/2, and horizontally by adistance 885 equaling PTC/2.

The pixels of array portion 811 capture values for the input image.Plus, in embodiments, additional values can be computed about the inputimage. The additional values could correspond to additional centerpoints 839, for the mathematical purposes of constructing the outputimage. For any one of the shown additional center points 839, additionalvalues can be computed for the same location with reference to moretiles in the array, even substantially all of the tiles.

FIG. 9 is a diagram of a two-layer offset stacked sensor array portion911. Array portion 911 includes an upper layer of green pixels 945, anda lower layer of red pixels 947 alternating with blue pixels 944.

In FIG. 9, pixel center points are not indicated separately, so as tonot clutter the drawing. If shown, these center points would be arrangedalong rows, such as rows 922, and along columns such as columns 924.Further, additional center points can be defined at the center points ofthe pixels of the lower layer, but are not shown separately. Thesecenter points are arranged along rows such as row 952, and along columnssuch as column 954.

In FIG. 9, the rectangular boundaries of the pixels define lines rotatedby 45° from the horizontal and the vertical. If one were to measurepixel pitch PTC along those rotated lines, then the effective pixelpitch measured along rows 922, 952 and columns 924, 954 would bedifferent. Pitch 962 between rows 922, and pitch 964 between columns 924would equal PTC*SQRT(2). And offsets 963, 965 would equal PTC/SQRT(2).

The pixels of array portion 911 capture values for the input image.Plus, in embodiments, additional values can be computed about the inputimage. The additional values could correspond to additional centerpoints 939, as above, for one or more tiles.

FIG. 10 is a diagram of a three-layer offset stacked sensor arrayportion 1011. Array portion 1011 includes an upper layer of green pixels1045, a middle layer of blue pixels 1044 and a lower layer of red pixels1047.

In FIG. 10, the center points of green pixels 1045 are arranged alongrows, such as rows 1022, and along columns, such as columns 1024.Distance 1062 equals distance 1064 equals pitch PTC. Blue pixels 1044are stacked over red pixels 1047 with no offset. However, blue pixels1044 are arranged along rows, such as row 1052, and along columns, suchas column 1054. The rows and columns of blue pixels 1044 have verticaland horizontal offsets of PTC/2 from rows 1022 and columns 1024.

The pixels of array portion 1011 capture values for the input image.Plus, in embodiments, additional values can be computed about the inputimage. The additional values could correspond to additional centerpoints 1039, as above, for one or more tiles.

FIG. 11 is a diagram of a three-layer offset stacked sensor arrayportion 1111. Array portion 1111 includes an upper layer of green pixels1145, a middle layer of blue pixels 1144 and a lower layer of red pixels1147. It will be recognized that array portion 1111 has a design similarto that of array portion 1011, except that it is rotated by 45°.

The pixels of array portion 1111 capture values for the input image.Plus, in embodiments, additional values can be computed about the inputimage. The additional values could correspond to additional centerpoints 1139, as above, for one or more tiles.

FIG. 12 is a diagram of a three-layer offset stacked sensor arrayportion 1211. Array portion 1211 includes an upper layer of green pixels1245, a middle layer of blue pixels 1244 and a lower layer of red pixels1247.

In FIG. 12, the center points of green pixels 1245 are arranged alongrows, such as rows 1222, and along columns, such as columns 1224.Distance 1262 equals distance 1264 equals pitch PTC. The middle layer isoffset from the upper layer vertically and horizontally by equaldistances of PTC/3, which is distance 1265. The lower layer is offsetfrom the middle layer vertically and horizontally by equal distances ofPTC/3, which is distance 1266.

The pixels of array portion 1211 capture values for the input image.Plus, in embodiments, additional values can be computed about the inputimage. The additional values could correspond to additional centerpoints 1239, as above, for one or more tiles.

The above description showed a variety of possible embodiments. Beforeproceeding with details of computing the additional values, an overallimaging system is now described.

FIG. 13 depicts a controller-based system 1300 for an imaging devicemade according to embodiments. System 1300 could be for the device ofFIG. 1.

System 1300 includes an image sensor 1310, which is made according toembodiments, such as by a pixel array. Examples of pixels arrays wereseen above. As such, system 1300 could be, without limitation, acomputer system, an imaging device, a camera system, a scanner, amachine vision system, a vehicle navigation system, a smart telephone, avideo telephone, a personal digital assistant (PDA), a mobile computer,a surveillance system, an auto focus system, a star tracker system, amotion detection system, an image stabilization system, a datacompression system for high-definition television, and so on.

System 1300 further includes a controller 1320, which is made accordingto embodiments. Controller 1320 could be the controller 120 of FIG. 1.Controller 1320 could be a Central Processing Unit (CPU), a digitalsignal processor, a microprocessor, a microcontroller, anapplication-specific integrated circuit (ASIC), a programmable logicdevice (PLD), and so on. In some embodiments, controller 1320communicates, over bus 1330, with image sensor 1310. In someembodiments, controller 1320 may be combined with image sensor 1310 in asingle integrated circuit. Controller 1320 controls and operates imagesensor 1310, by transmitting control signals from output ports, and soon, as will be understood by those skilled in the art.

Controller 1320 may further communicate with other devices in system1300. One such other device could be a memory 1340, which could be aRandom Access Memory (RAM) or a Read Only Memory (ROM), or acombination. Memory 1340 may be configured to store instructions to beread and executed by controller 1320. Memory 1340 may be configured tostore images captured by image sensor 1310, both for short term and longterm.

System 1300 could further include an image processor 1325, forprocessing the input image according to embodiments. Image processor1325 could be image processor 125 of FIG. 1. In some embodiments, imageprocessor 1325 communicates, over bus 1330, with image sensor 1310.

Another such device could be an external drive 1350, which can be acompact disk (CD) drive, a thumb drive, and so on. One more such devicecould be an input/output (I/O) device 1360 for a user, such as a keypad,a keyboard, and a display. Memory 1340 may be configured to store userdata that is accessible to a user via the I/O device 1360.

An additional such device could be an interface 1370. System 1300 mayuse interface 1370 to transmit data to or receive data from acommunication network. The transmission can be via wires, for examplevia cables, or USB interface. Alternately, the communication network canbe wireless, and interface 1370 can be wireless and include, forexample, an antenna, a wireless transceiver and so on. The communicationinterface protocol can be that of a communication system such as CDMA,GSM, NADC, E-TDMA, WCDMA, CDMA2000, Wi-Fi, Muni Wi-Fi, Bluetooth, DECT,Wireless USB, Flash-OFDM, IEEE 802.20, GPRS, iBurst, WiBro, WiMAX,WiMAX-Advanced, UMTS-TDD, HSPA, EVDO, LTE-Advanced, MMDS, and so on.

One more such device can be a display 1380. Display 1380 could bedisplay 180 of FIG. 1. Display 1380 can show to a user a tentative imagethat is received by image sensor 1310, so to help them align the device,perhaps adjust imaging parameters, and so on.

Computations according to embodiments are now described in more detailfor a sample array. Computations according to embodiments will besimilarly possible for different arrays. The following description is interms of a journey starting from physical considerations, and graduallyproceeding to more abstract, mathematical considerations.

FIG. 14 is a diagram of a physical two-layer offset stacked sensor arrayportion 1414. It will be recognized that a special case of array portion1414 is array portion 811. More particularly, array portion 1414includes an upper layer of pixels and a lower layer of pixels. Thepixels of array portion 1414 have pixel center points 1436, indicated asfull dots. Pixel center points 1436 may correspond to locations ofcaptured values. Pixel center points 1436 are in the same plane, whichis the plane of the drawing, regardless of whether they were contributedby pixels of the upper layer or of the lower layer.

The pixels of array portion 1414 capture values for the input image.Plus, in embodiments, one or more additional values can be computedabout the input image. In FIG. 14, an additional center point 1439 isshown at an advantageously chosen location, namely at the intersectionof one of rows 1422 of the top pixel layer, with one of columns 1454 ofthe bottom pixel layer. While only one additional center point 1439 isshown, more can be chosen, as per the above. The additional values couldcorrespond to the one or more additional center points 1439, for themathematical purposes of constructing the output image.

As the starting point of the journey, array portion 1414 is also calleda pixel layer location grid, and is made from a top grid and a bottomgrid. The center points of the pixels in the upper layer are disposedalong rows 1422 and columns 1424, which thus comprise the top grid. Rows1422 have a vertical pixel pitch given by distance 1462. Columns 1424have a horizontal pixel pitch given by distance 1484. In thisembodiment, the vertical pitch equals the horizontal pitch, equals PTC.

The center points of the pixels in the lower layer are disposed alongrows, such as rows 1452, and columns, such as columns 1454. These rowsand these columns thus comprise the bottom grid. In embodiments of FIG.8, the rows and the columns in the lower layer have the same pitch PTCas those in the upper layer. The lower layer is offset from the upperlayer, vertically by a distance 1463 equaling PTC/2, and horizontally bya distance 1485 equaling again PTC/2.

FIG. 15 is a diagram of a portion 1515 of a physical sampling grid.Portion 1515 is made only from the rows and the columns of the pixellayer location grid of FIG. 14, plus pixel center points 1436. It willbe appreciated that the top grid of FIG. 14 is superimposed on thebottom grid of FIG. 14, and the top grid and the bottom grid have beencollapsed and merged into a single plane in FIG. 15. Portion 1515 thusrepresents the locations of all the physical sampling photo-sites in thearray, which will provide the captured values.

In FIG. 15, an additional value can be computed about the input image,which would correspond to an additional center point 1439. The selectionof the array of FIG. 14 permits more such points, as is now described.

FIG. 16 is a diagram of a portion 1616 of a sample image constructiongrid according to embodiments. Portion 1616 is made from a physicalsample image construction grid, such as that of FIG. 15, together withadditional center points 1639 that will be used. It will be recognizedthat additional center points 1639 are created by replicating additionalcenter points 1439 of FIG. 14 in advantageous spaces of the physicalsample image construction grid portion lower layer of FIG. 15.

As such, in sample image construction grid portion 1616 everyintersection of a row and a column will contribute a value for the finalimage. Values corresponding to pixel center points 1436 will have beenthe captured values for the input image, and values corresponding toadditional center points 1639 will have been the additional computedvalues about the input image.

It will be further appreciated that, since every intersection in FIG. 16will contribute a value for the final image, the effective pitch willbecome less. To better appreciate this, however, a more detaileddescription is now presented as to how a tile is defined.

A tile is a group of pixels that may be adjacent to each other in avertical dimension, and/or a horizontal dimension and/or a depthdimension. Plus, the pixels of a tile are a minimum in number. Moreover,taken as a group, the pixels of a tile have substantially identicalproperties, regardless of their location within the array. Thedefinition of a tile is used when there are pixels of different colors,which are expected to work in combination. Sometimes a process isapplied for this combination, called demosaicking, as it treats thepixels of a tile as a mosaic. As will be appreciated in consideringthese concepts, tiles can also be defined from the pixel center points.

In some embodiments, for each tile, an additional center point may bereplicated along with a tile. While in the examples below an additionalcenter point is shown within a tile, such is not necessary.

In some embodiments, the additional values that correspond to additionalcenter points are demosaicked jointly with the captured values. Theoutput image can then be constructed from the joint demosaicking.

Examples of such tiles are now described. Through these examples, itshould be kept in mind that, in other embodiments, two or moreadditional center points can be defined for each tile.

FIG. 17 is a diagram showing a single-layer sensor array portion 1711.Portion 1711 is of the same Bayer pattern as in FIG. 5. A sample tile is1780. A sample additional center point 1739 is shown within tile 1780.

FIG. 18 is a diagram showing a three-layer stacked sensor array portion1811. Portion 1811 is of the same pattern as in FIG. 7, with the sametypes of pixels. A sample tile is 1880, of three pixels adjacent only inthe depth dimension. A sample additional center point 1839 is shownwithin tile 1880.

FIG. 19 is a diagram showing a two-layer stacked sensor array portion1911. Portion 1911 is of the same pattern as in FIG. 6, with the sametypes of pixels. A sample tile is 1980, of four pixels adjacent in thewidth and in the depth dimension. Two sample additional center points1939 are shown within tile 1980.

FIG. 20 is a diagram showing a two-layer stacked offset sensor arrayportion 2011. Portion 2011 is of the same pattern as in FIG. 8, with thesame types of pixels. A sample tile is 2080, of four pixels adjacent inthe width and in the depth dimension with offset. A sample additionalcenter point 2039 is shown within tile 2080. More such additional centerpoints are possible. An example is now described.

FIG. 21 shows sequential diagrams depicting another journey. Arrayportion 2011 is repeated from FIG. 20, with tile 2080 designated.

Moving downward from array portion 2011, the pixels of tile 2080 aloneare then considered isolated. In addition, locations 2140 are shown,where additional values will be computed. It should be noted that, forsome of these locations 2140, there is already a captured value of onecolor from the underlying color pixel. The hexagons with embeddedquestion marks for locations 2140 are a vehicle for showing whereadditional values could be computed.

Moving further downward from tile 2080, an image construction grid 2116is shown. Points P1, P2, . . . , P8 are shown at the intersections ofgrid 2116. All these points P1, P2, . . . , P8 are additional centerpoints 2139, and therefore are shown as including a circle 2139.Additional values will be computed that correspond to them. Plus, pointsP1, P3, P6, P8 are also pixel center points, and therefore are shown asincluding a dot 2136. A captured value corresponds to each of them, fromthe underlying pixel. It will be observed that points P1, P3, P6, P8include both a dot 2136 and a circle 2139. Notwithstanding the capturedvalue, an additional value maybe calculated for these points, such asfor a different color.

For the computation of the additional values, according to embodiments,one or more filter functions can be used. Examples are now described. Asthe description proceeds, it will be seen that a filter function canhave filter parameters, and different values for the filter parametersmay be used.

FIG. 22 shows sequential diagrams. Array portion 1414 is repeated fromFIG. 14. Within array portion 1414, a pixel center point 2236 isdesignated. Moreover, an additional value will be computed for pixelcenter point 2236. As such, pixel center point 2236 is also designatedas an additional center point 2239.

According to embodiments, the additional value may be computed from akernel of the captured values that surround additional center point2239. Accordingly, a kernel 2277 can be defined around additional centerpoint 2239, which encircles the captured values that will be used forthe computation of the additional value. It will be observed that kernel2277 also encircles points like point 2237, which have no capturedvalue.

Moving downward from array portion 1414, a location set 2282 is shownfor kernel 2277. Location set 2282 includes the captured values forpixel center points Q1, Q2, . . . , Q13. Pixel center points Q1, Q2, . .. , Q13 correspond to enumerated locations. Location set 2282 furtherincludes null values 2230, for points like point 2237, which have nocaptured value. Those can be ignored or set to zero, when the additionalvalue is computed.

As mentioned above, the process of FIG. 22 is intended for computing theadditional value at additional center point 2239. The process may needto be repeated for computing additional values at additional centerpoints of a tile, such as those shown in image construction grid 2116 ofFIG. 21 for tile 2080. Location set 2282 can also be thought of as akernel and/or a vector Hp. In that context, additional center point 2239is also defined as a unique filter location p, and location set 2282shows the locations within kernel Hp, centered over a constructionlocation p, within tile (i, j), and layer l. For example, centrallocation Q7 of location set 2282 could be applied for point P6 of imageconstruction grid 2116, and then for point P8. That will also be aninstance where the additional value that corresponds to, say, additionalcenter point P6 within tile 2080 is computed from one of the capturedvalues within tile 2080, such as the value at P3, and from one of thecaptured values outside the certain tile, such as a value one abovepoint P2.

FIG. 23 shows sequential diagrams. Array portion 1414 is repeated fromFIG. 14. Within array portion 1414, pixel center points 2336 aredesignated. Captured values correspond to pixel center points 2336.Moreover, an additional value will be computed that will correspond toadditional center point 2339. Differently, additional center point 2339is not at the intersection of any axes, and does not enjoy the manysymmetries of point 2239 in previous diagrams.

According to embodiments, a kernel 2377 can be defined around additionalcenter point 2339, which encircles the captured values that will be usedfor the computation of the additional value. As also above, kernel 2377also encircles points that have no captured value, and which are notdesignated separately in FIG. 23.

Moving downward from array portion 1414, a location set 2382 is shownfor kernel 2377. Location set 2382 includes the captured values forpixel center points S1, S2, . . . , S12. Pixel center points S1, S2, . .. , S12 correspond to enumerated locations. Point 2339 is also shown.

From the above, some filter parameters can be appreciated. First afilter parameter can be the size of kernels 2277, 2377, in other wordsthe size of the filter. Good values seem to be 5×5 to 7×7. Good valuesfor rectangular-shaped kernels further seem to be those using an oddnumber by the same odd number, so that the filter will be evenly weighedaround the additional center point. Of course, a kernel according toembodiments need not be rectangular. Second, a filter parameter can bethe weight coefficients that can multiply the different captured values,for example depending on their location within the kernel.

More detailed computation methods according to embodiments are nowdescribed, for the additional values. These additional values can bedenominated in any suitable color space. Embodiments may include the RGBcolor space, the red-white-blue color space and theluminance—chrominance color space. More particular examples are nowdescribed.

FIG. 24 shows equations that can be used for computing R, G, B valuesaccording to embodiments. It will be recognized that these equationsrepresent the process of linear convolution, applied in two dimensions,by summing captured values of pixels that have been specially weightedaccording to the above mentioned weight coefficients. The convolution isin a neighborhood around the construction location.

In the equations of FIG. 24, subscript l is the number of layers in thepixels sensor array. The convolution is a summation over all layers l.

Each tile T can have identifying subscripts T(i, j), and the convolutionmay be performed for locations p within each tile (i,j). VectorX(i,j,p,l) stands for the captured values within a kernel at tileT(i,j), intra-tile location p, and layer l.

The weight coefficients, which are also known as weights, can be givenby vector filter H. There can be multiple such filters H, one for eachcombination of layer l, intra-tile construction location p, and color.For the Red, Green, Blue colors, these filters can be H(p,l)^(R),H(p,l)^(G) and H(p,l)^(B). Each such filter can be of a suitable size,such as the previously mentioned 7×7.

The values for R, G, B can therefore be computed from FIG. 24. It willbe appreciated that locations within kernel H and corresponding where aphysical pixel does not exist can be ignored, because no captured valueis provided. Ignoring can be implemented by excluding such from thecalculations, setting the appropriate weight coefficient to zero, etc.

Alternative computations may be performed by using theluminance—chrominance color space. If these are used, then they may befurther converted to R, G, B, in which case fewer calculations may beneeded than the above-described direct R, G, B computations. Particularembodiments are now described.

FIG. 25A shows an equation for estimating luminance Y(i,j,p) for theadditional values. A different filter can be used, namely H(p,l)^(Y),which can be on the larger side, such as 7×7, given the nature ofluminance. Again, summation can be over all layers l.

FIG. 25B shows an equation for estimating a color difference C(i,j,p,l).The color difference is between the captured or sensed values, and whatis derived by the luminance computation of FIG. 25A.

FIG. 25C shows equations for estimating chrominance values C(i,j,p)^(R),C(i,j,p)^(G) and C(i,j,p)^(B) for the R, G, B colors respectively. Theestimating uses the color difference estimated from the equation of FIG.25B. Different filters can be used, namely H(p,l)^(CR), H(p,l)^(CG) andH(p,l)^(CB), which can be on the smaller side, such as 5×5. Again,summation is over all layers l.

FIG. 25D shows equations for estimating RGB values, such as those ofFIG. 24. In FIG. 25D, however, the RGB values are reconstructed from theluminance values of FIG. 25A and the differences in chrominance fromFIG. 25C.

Of course, the above described operations assume that the one or morefilter functions, or just filters, are known for the computations.According to embodiments, the filters are first trained, to determinegood values for their parameters. The trained filters are then used forready devices. Examples are now described.

FIG. 26 is a diagram 2600 for describing filter training operationsaccording to embodiments. A reference image 2610 can be selected, and istreated as the source. Reference image 2610 can be selected from acollection of typical scenes, both natural and man-made. In preferredembodiments, the frequency of types of scenes in the selected imagereflects the frequency of those types of scenes captured byrepresentative users of imaging device 100, such as photographers.

A target image 2612 can be derived from selected reference image 2610according to an embodiment of one of a number of possible derivationmethods. Sample algorithms for such derivation methods are describedlater in this document. Target image 2612 can also be thought of as atarget version of reference image 2610, and also be called targetreference image. Target image 2612 corresponds to how it is desired tohave reference image 2610 constructed by embodiments.

A set of RGB values 2614 is known about reference image 2610. Accordingto a process 2635, which simulates processes according to embodiments,an output test image 2640 is constructed from set of values 2614, andusing tentative values for the parameter of the filter function. Process2635 may include computing additional test values and demosaicking. Theadditional test values can be demosaicked jointly with the input testvalues. Output test image 2640 can be constructed from the jointdemosaicking.

Output test image 2640 may then be compared to target image 2612, forhow close they are, i.e. how well they match. The process may berepeated a number of times, using different parameter values, until theyare satisfactorily close. This way, filters are trained, or learned, byminimizing an error, i.e. a difference between constructed output testimage 2640 and target image 2612. Sample algorithms for minimizing theerror are described later in this document.

In some embodiments, the input test values can be determined by actuallyexposing a test array to the reference image, and thus capturing animage. The reference image may be a scene printed on a test chart.Preferably, the image of the scene is not overexposed and has low noise.

In such cases, target image 2612 can be obtained by aligning referenceimage 2610 to the captured image as precisely as possible, anticipatingpossible geometric distortions that may be present in the test pixelarray. Then target image 2612 can be calculated.

In other embodiments, the input test values can be determined bycalculating an expected response of the pixels due to exposure to thereference image. This can be implemented in any number of ways.

One such way is to determine a Point Spread Function (PSF). PSF can bemeasured for a test pixel array. Alternately, a Modulation TransferFunction (MTF) can be measured, to obtain the PSF. If measurement is notpossible, the PSF could be estimated for example using lens parametersand known or target sensor PSF, possibly obtained by system simulations.

In such cases, target image 2612 can be simulated by calculatingexpected response on the construction grid. For example, ahigh-resolution bitmap image of a natural scene can be taken as thereference image, which is a faithful linear representation of thecaptured scene. The noise could be low, there could be few or nooverexposed areas where RGB values are clipped, and no over-sharpening.If the image had been gamma-corrected, it could be linearized byapplying reverse gamma transform. In addition, reverse color correctioncan be applied, which would correspond to the image's color space. Theexpected responses at physical photo-sites can be calculated. This canbe achieved by applying the PSF obtained from the previous step. Thedesired responses can be calculated after demosaicking on the imageconstruction grid, to obtain the target image 2612.

FIG. 27 shows a flowchart 2700 for describing filter training methodsaccording to embodiments. The methods of flowchart 2700 may also bepracticed by embodiments described above. Or, one or more of the filterfunctions may have been trained by a separate processor that performsactions. These actions may result in operations such as the below.

According to an operation 2710, input test values can be received for areference image. The reference image can be reference image 2610, andinput test values can be set 2614. The input test values may bedetermined in any number of ways.

According to another operation 2720, a tentative value is selected for aparameter of the filter function. If this is the first time operation2720 is performed, the value may be selected by being set tentatively.If this is a subsequent time operation 2720 is performed, the value maybe adjusted from what was set tentatively. Adjustment may be further inview of a determination of error in operation 2750.

According to another operation 2730, additional test values can becomputed from the input test values about the reference image. Theadditional test values can be computed using the last time that thetentative value of operation 2720 was executed. In addition,demosaicking can be performed as per the above.

According to another operation 2750, a target version of the referenceimage may be compared with an output test image. The output test imagemay have been constructed from the input test values and from theadditional test values, such as from operation 2730. Comparing may beperformed to see how well there is matching. The extent of matching maybe determined by quantifying a difference as an error, as mentionedabove, and then trying to minimize the error. If matching is notsatisfactory, the execution may continue to operation 2730, and theparameter value may be further adjusted based on the comparison.

Embodiments for minimizing the error are now described.

FIG. 28 shows equations that can be used for training filters forpreparing an image with RGB values according to embodiments. In thesecases, training will be for direct RGB construction.

Equations 2810 are an over-determined system of linear equations.Training involves error minimization in equations 2810. The values ofR*, G*, B* are values from the target image. The vectors shown with aprime, such as H′, are the transpose vectors from what would be withoutthe prime, such as H.

The challenge with equations 2810 is that the unknown of interest is H′,which is not given analytically. The challenge is met by writingequation 2820 as a synoptic representation of each of equations 2810.Then, equation 2830 displays a way to solve equation 2820 for the valueof A. As such, the form of equation 2830 will be used for solvingequations 2810 for H′.

Alternately, the training might be solved with fewer computations withembodiments that first compute the luminance chrominance. The below usesconcepts already described above with reference to FIGS. 25A-25D.

FIGS. 29A, 29B, 29C, 29D show equations that can be used for trainingfilters first in luminance and chrominance values according toembodiments. The equation of FIG. 29A gives the value of the luminance,at least for a target reference image. The equation of FIG. 29B presentsthe over-constrained problem that includes the transpose of theluminance filter H′(p,l)^(Y). Then the luminance filter H(p,l)^(Y) canbe solved, for the filter training. Solving can be done using numericalmethods, such as least mean square error.

Then the chrominance filters H(p,l)^(CR), H(p,l)^(CG) and H(p,l)^(CB)can be solved for the filter training. The equations of FIG. 29C aresimilar to those in FIGS. 25A, 25B, and 25D. The equations of FIG. 29Cpresent the same over-constrained problem as Equations 2810 of FIG. 28,and can be solved similarly.

Additional image operations are now described.

FIG. 30 is a diagram 3077 illustrating sample image operations. An inputimage 3014 may undergo one or more operations 3070, after which theyrender an output image 3040. A number of such operations are possibleaccording to embodiments. For example an image can be resized, alongwith being demosaicked. An example is now described.

FIG. 31 is a diagram illustrating resizing. A portion 3111 of an arrayis shown, which is a portion of array 811 of FIG. 8. Portion 3111 alsoshows unit tile 2080, horizontal pixel pitch 884, and horizontal offsetdistance 885 that is half the size of horizontal pixel pitch 884.

According to a resizing operation 3170, the image can be resized into anew sample image construction grid 3116. Image construction grid 3116defines locations, marked by dots, at the intersections of new rows 3172and columns 3174. These new rows and columns give image constructiongrid 3116 its own pitch, which can be called construction pitch. Theconstruction pitch can be different than the pixel pitch. Being resized,image construction grid 3116 will no longer map into the physical pixelsthe same way.

More particularly, in portion 3111 the pixel center points aredistributed along a certain direction, according to a pixel pitch 884.In FIG. 31 the certain direction is the horizontal direction, but onecould be describing equivalently the vertical direction. Plus, in imageconstruction grid 3116, at least some of the additional center pointscan be distributed along the certain direction according to theconstruction pitch. Here, the certain direction being the horizontaldirection, a construction pitch 3194 in the horizontal direction islarger than physical sampling grid pitch 885. This is indicated bycontrasting indication 3113, which contrasts a size of twice pitch 885,namely pixel pitch 884, with a size of twice construction pitch 3194.

In the example of FIG. 31, a new unit tile 3180 is shown, which is alsocalled the resized tile and elementary resized tile. It should be notedthat resized tile 3180 is not defined from tile 2080, but it becomesdefined from the minimum number of locations or positions that areuniquely positioned on construction grid 3116 with respect to thephysical pixel photo-sites. Tile 3180 has nine such locations orpositions, shown in dots, for which filter values will need to becomputed. Only four locations S1, S6, S8, S9 of the nine are designated,so as not to clutter the drawing.

The expansion from the input image to the output image can be quantifiedin terms of the horizontal scale ratio, of the horizontal constructionpitch divided by the horizontal physical pitch, and the vertical scaleratio, of the vertical construction pitch divided by the verticalphysical pitch. The total scale ratio would be the horizontal scaleratio times the vertical scale ratio. A sample such total scale ratiocan be about 1.5×.

In fact, in embodiments of resizing, there can be expansion orcontraction in the horizontal and the vertical direction concurrently.In the particular case of embodiments, expansion is advantageouslypossible because additional values can be computed between the capturedvalues. The output image can be resized, in some embodiments, to matchthe increase in limiting resolution, which can be about 20%.

In embodiments such as that of FIG. 31, where image construction grid3116 is chosen such that there is a green pixel photo-site at eachlocation, the total scale ratio corresponds to the number of samples inoutput image/the number of green pixels in the input image. Accordingly,a 2 Megapixel camera could effectively yield an image of a 3 Megapixelcamera consistently, with a total scale ratio of about 1.5×. A personskilled in the art will determine what further choices of a constructionpitch are advantageous for resizing, given any image construction grid.

Moreover, for each filter location, a kernel will have to be defined aspreviously for each unique filter location. A filter kernel on imageconstruction grid 3116 is defined around each construction location onimage construction grid 3116. It will be observed that values will haveto be computed for kernel locations that are not on the physical pixelgrid, for example as was indicated in FIG. 23 by additional center point2339. Filter coefficient locations within kernel H may be enumerated.Coefficients at locations where a physical pixel center is not presentcan be ignored or set to 0.

The kernel size could be larger, when resize is applied. The larger sizemay result from trying to match the original kernel size without resize,so as to avoid color aliasing. For example, if 2× downsize is appliedhorizontally and vertically, the new kernel size with resize could cover2× more photo-site samples horizontally and vertically. The rest of thealgorithm could be as before.

Additional image operations could be implemented, optionally as a singleoperation with operations described above, as was true for the resizingoperation of FIG. 30. Such additional image operations include colorcorrection, denoise and sharpening. These additional image operationsalso include color space conversion, for example by outputting YUV bylinear conversion from RGB, or in reverse. These additional imageoperations moreover include bad pixel correction, where a sample isreconstructed when missing. Bad pixel correction can then be performedusing properly trained kernels. Sample embodiments are now described.

In embodiments, once filters are implemented, further implementing theseadditional operations as a single operation using the filters mayadvantageously remove the need for including separate processing stagesfor these additional operations. Indeed, these additional operations canbe implemented as variations in the values of the already implementedfilters, from what these values would be. Moreover, such can be evencustomized to particular components of an imaging device, such asimaging device 100.

FIG. 32 is a diagram 3200 for describing filter training operationsaccording to embodiments, similarly with those of FIG. 26. A referenceimage 3210 can be selected, similarly with reference image 2610.

A target image 3212 can derived from selected reference image 3210.Target image 3212 can also be thought of as a target version ofreference image 3210. Target image 3212 corresponds to how it is desiredto have reference image 3210 constructed by embodiments. Derivation canbe twofold. Initial derivation can be as described earlier in thisdocument. Beyond that, derivation includes applying one or more desiredadditional image operations 3288 on reference image 3210, so as toderive target image 3212. Additional image operation 3288 can includeone or more of color correction, denoise, sharpening, color spaceconversion, bad pixel correction, resize, etc.

According to a process 3235, which simulates processes according toembodiments, an output test image 3240 is constructed from set of values3214, and using tentative values for the parameter of the filterfunction. Process 3235 may include computing additional test values anddemosaicking, similarly with process 2625.

Output test image 3240 may then be compared to target image 3212, forhow close they are. The process may be repeated a number of times, usingdifferent parameter values, until they are satisfactorily close. Thisway, filters are trained, or learned, by minimizing the error betweenconstructed output test image 3240 and target image 3212, as above.Accordingly, the filters within process 3235 will be trained to performalso desired additional image operation 3288.

Customizations can be further added to additional image operation 3288according to embodiments. For example, there can be customization forconstruction vs. pixel array location, lens zoom position, focusposition. As such, customization could be for each manufactured cameramodule for calibration purposes; such camera modules could provide thetest arrays. A camera PSF/MTF varies with respect to pixel arraylocation (center or periphery), lens zoom and focus potion, individualmodule characteristics, etc. These customizations can help attain betterimage quality, by considering and compensating these PSF/MTF variations.Additional sets of filters can be trained for each configuration, whileonly one set of filters could be applied for the final image each time.

Additional image operation 3288 can have further uses according toembodiments. Filters can be trained, so that other color configurationscan also be utilized. These other color configurations can includescenarios with other than three color channels, such as monochrome, RGBW(RGB+white), etc.

A challenge is that, although color correction can be combined withdemosaicking (and sharpening), sometimes color correction requiresnon-linear processing. As such, color correction sometimes cannot bedone jointly with demosaicking. In cases when sharpening is appliedafter color correction, sharpening could be disjoint from demosaicking,and more stages may be required.

FIG. 33 is a diagram illustrating a specific composite image operation3300. A joint demosaicking and sharpening signal operation 3305 isapplied to an input image 3310. One of the results is a reconstructedimage 3340 that includes RGB values, and another is a sharpening signal3382 that includes S values indicating a magnitude of the sharpeningsignal. More particularly, a filter may be added for each location. Thisfilter can be trained to generate a sharpening signal by providing a(simulated) target image for sharpening signal. As such the sharpeningsignal is added to image after color correction to obtain the sharpenedimage in this embodiment. A color correction operation 3342 can then beapplied to reconstructed image 3340, and then a sharpening operation3384 applies sharpening signal 3382, so that reconstructed image 3340becomes final output image 3386.

In the methods described above, each operation can be performed as anaffirmative step of doing, or causing to happen, what is written thatcan take place. Such doing or causing to happen can be by the wholesystem or device, or just one or more components of it. In addition, theorder of operations is not constrained to what is shown, and differentorders may be possible according to different embodiments. Moreover, incertain embodiments, new operations may be added, or individualoperations may be modified or deleted. The added operations can be, forexample, from what is mentioned while primarily describing a differentsystem, device or method.

This description includes one or more examples, but that does not limithow the invention may be practiced. Indeed, examples or embodiments ofthe invention may be practiced according to what is described, or yetdifferently, and also in conjunction with other present or futuretechnologies.

Reference to any prior art in this specification is not, and should notbe taken as, an acknowledgement or any form of suggestion that thisprior art forms parts of the common general knowledge in any country.

A person skilled in the art will be able to practice the presentinvention in view of this description, which is to be taken as a whole.Details have been included to provide a thorough understanding. In otherinstances, well-known aspects have not been described, in order to notobscure unnecessarily the present invention.

Other embodiments include combinations and sub-combinations of featuresdescribed herein, including for example, embodiments that are equivalentto: providing or applying a feature in a different order than in adescribed embodiment; extracting an individual feature from oneembodiment and inserting such feature into another embodiment; removingone or more features from an embodiment; or both removing a feature froman embodiment and adding a feature extracted from another embodiment,while providing the advantages of the features incorporated in suchcombinations and sub-combinations.

The following claims define certain combinations and subcombinations ofelements, features and steps or operations, which are regarded as noveland non-obvious. Additional claims for other such combinations andsubcombinations may be presented in this or a related document.

1. An imaging device, comprising: an array having a set of pixelsconfigured to capture respective values for an input image, the capturedvalues corresponding to pixel center points of the respective pixelsthat captured the values; a processor configured to compute, from thecaptured values, additional values about the input image, at least someof the additional values corresponding to additional center pointsdifferent from any of the pixel center points; and one or more memoriesconfigured to store the captured values and the additional values. 2.The device of claim 1, in which the additional values are further mergedwith the captured values when stored.
 3. The device of claim 1, furthercomprising: a display configured to display an output image constructedfrom both the captured values and from the additional values.
 4. Thedevice of claim 3, in which the additional values are demosaickedjointly with the captured values, and the output image is constructedfrom the joint demosaicking.
 5. The device of claim 1, in whichsubstantially identical tiles are defined from the pixel center points,and one or more of the additional values that corresponds to one of theadditional center points within a certain ones of the tiles is computedfrom one of the captured values within the certain tile and from one ofthe captured values outside the certain tile.
 6. The device of claim 1,in which a sum of a number of the captured values plus a number of theadditional values is larger than a number of the pixels in the array. 7.The device of claim 1, in which the pixels of the array are arranged inat least a lower layer and in an upper layer that is vertically stackedover the lower layer, and the pixel center points are defined as thosefor the pixels in the upper layer plus those for projections of thepixels in the lower layer onto a plane of the upper layer.
 8. The deviceof claim 1, further comprising: a display configured to display anoutput image constructed from both the captured values and from theadditional values.
 9. The device of claim 8, in which the additionalvalues are demosaicked jointly with the captured values, and the outputimage is constructed from the joint demosaicking.
 10. The device ofclaim 1, in which the additional values are computed using a filterfunction.
 11. The device of claim 10, in which the filter function hasbeen trained by a processor performing actions resulting in operationscomprising: receiving input test values for a reference image; selectinga tentative value for a parameter of the filter function; computing,from the input test values, additional test values about the referenceimage using the tentative value; and adjusting the parameter value. 12.The device of claim 11, in which the input test values have beendetermined by exposing a test pixel array to the reference image. 13.The device of claim 11, in which the input test values have beendetermined by calculating an expected response of the pixels due toexposure the reference image.
 14. The device of claim 11, in which theparameter of the filter function is one of a size of the filter, and aweight coefficient of the filter.
 15. The device of claim 11, in which atarget version of the reference image is compared with an output testimage constructed from the input test values and from the additionaltest values, and the parameter value is adjusted based on thecomparison.
 16. The device of claim 15, in which the additional testvalues are demosaicked jointly with the input test values, and theoutput test image is constructed from the joint demosaicking.
 17. Thedevice of claim 15, in which an additional image operation has beenperformed on the reference image to derive the target version of thereference image.
 18. The device of claim 1, in which the pixel centerpoints are distributed along a certain direction according to a pixelpitch, and at least some of the additional center points are distributedalong the certain direction according to a construction pitch differentthan the pixel pitch.
 19. An imaging device, comprising: an array havinga set of pixels configured to capture respective values for an inputimage, the captured values corresponding to pixel center points of therespective pixels that captured the values; a processor configured tocompute, from the captured values, additional values about the inputimage, at least some of the additional values corresponding toadditional center points different from the pixel center points; and adisplay configured to display an output image constructed from thecaptured values and from the additional values. 20-32. (canceled)
 33. Acomputer comprising: a processor and a non-transitory storage mediumcoupled with the processor, the storage medium storing one or moreprograms which, when executed by the processor, result in: receivingcaptured values for an input image, the captured values corresponding topixel center points of respective pixels that captured the values;computing, from the captured values, additional values about the inputimage, at least some of the additional values corresponding toadditional center points different from the pixel center points; andcausing the captured values and the additional values to be stored inone or more memories. 34-44. (canceled)
 45. A method, comprising:receiving captured values for an input image, the captured valuescorresponding to pixel center points of respective pixels that capturedthe values; computing, from the captured values, additional values aboutthe input image, at least some of the additional values corresponding toadditional center points different from the pixel center points; andcausing the captured values and the additional values to be stored inone or more memories. 46-56. (canceled)
 57. A method for an imagingdevice having an array of pixels at respective pixel center points ofthe array, a processor and a display, the method comprising: capturingvalues for an input image in a set of the pixels, the captured valuescorresponding to pixel center points of the respective pixels thatcaptured the values; computing, from the captured values, additionalvalues about the input image, at least some of the additional valuescorresponding to additional center points different from the pixelcenter points; and displaying an output image constructed from thecaptured values and from the additional values. 58-68. (canceled)
 69. Amethod for an imaging device having an array of pixels at respectivepixel center points of the array, a processor and one or more memories,the method comprising: capturing values for elements of an input imagein a set of the pixels, the captured values corresponding to pixelcenter points of the respective pixels that captured the values;computing, from the captured values, additional values about the inputimage by the processor, at least some of the additional valuescorresponding to additional center points different from the pixelcenter points; and storing the captured values and the additional valuesin the one or more memories. 70-80. (canceled)